AR Technologies        
                                                        Compiled By Alem Fitwi
                                                        Sunnyvale, CA, USA
                                                        April 2022

                                                        Src: Internet, Optics Conferences & Old Notes

from IPython.display import Image

0. Projected VA and AR Evolution

• ARVR Community's Vision: Moving from gaming to everyday life in three major steps

2015-2020

• VR is mostly perceived as a gaming and entertainment tool in the first five years

2020-2025

• At this stage, VR & AR are expected to merge together, where VR will grow to become a more promising professional platform.

2025-2030

• Advanced VR sets are expected to be created, capable of reading our minds and helping us get our works done on the go in any situation. It is projected that it would be hard to live without them.

1. Basics Of Waveguides (WG)

• Typical Electromagnetic Waves (EMW) Behavior: Light waves across the electromagnetic spectrum behave in similar ways. Depending on the composition of the object and the wavelength of the ligh, when the light wave encounters an object, it is either:
  • transmitted: propagated from $T_x$ to $R_x$,
  • reflected: Reflection is when incident light (incoming light) hits an object and bounces off. Very smooth surfaces such as mirrors reflect almost all incident light,
  • absorbed: Absorption occurs when photons from incident light hit atoms and molecules and cause them to vibrate,
  • refracted: Refraction is when light waves change direction as they pass from one medium to another. Light travels slower in air than in a vacuum, and even slower in water. As light travels into a different medium, the change in speed bends the light. Different wavelengths of light are slowed at different rates, which causes them to bend at different angles. ,
  • polarized: An expression of the orientation of the lines of electric flux in an electromagnetic field (EM field),
  • diffracted: Diffraction is the bending and spreading of waves around an obstacle. It is most pronounced when a light wave strikes an object with a size comparable to its own wavelength. Spectrometer uses diffraction to separate light into a range of wavelengths, or
  • scattered: Scattering occurs when light bounces off an object in a variety of directions. The amount of scattering that takes place depends on the wavelength of the light and the size and structure of the object.
• Waveguide(WG) is a structure that guides waves (like electromagnetic waves (EMW) or sound) with minimal loss of energy by restricting the transmission of energy to uni-direction.

$$T_x\ \rightarrow\ waveguide\ \rightarrow\ R_x$$

Image("./Figs/wg1.png")

        Fig 1: Propagation of EMWs Via A WG

• In signal Transmission, attenuation is a serious problem and that is the main reason why we need to use WG for transmitting signals or EMWs from the sender/source end to receiver/sink end. $$Attenuation \rightarrow \frac{Area\ of\ Receiver}{Area\ of\ Propagated\ waves}$$
  • Attenuation is mesure db/mm, db/cm, db/m, db/km, ...
• WG's are really amazing and hugely solve the problem of Attenuation
• WG can be often treated as a Linear System.
  • Output wave can be figured out if the ff are known:
    1. Transfer Function of the WG $\rightarrow$ $FFT[f(t)]$
    2. Shape of the Wave
• WG's could be one of the following types depending on where and how they are employed:
  • RF Waveguides: An electromagnetic feed line used for high radio frequency (RF) microwave signals in high-power $T_xs$ and $R_xs$, commonly used in radar equipment, microwave ovens, & satellite dishes
  • Acoustic Waveguides: A physical structure for guiding sound waves. Sound in an acoustic waveguide behaves like EMWs on a transmission line. Waves on a string, like the ones in a tin can telephone, are a simple example of an acoustic waveguide.
  • Optical Waveguides:A physical structure that guides EMW in the optical spectrum. Common examples are Optical Fiber Waveguides, Trasparent Dielectric Waveguides made of plastic and glass, liqui light guides, and liquid waveguides. They all help propagate light rays from one end to the other.
• Optical Waveguides:
  • Geometrical Classification:
    • Planar
    • Strip
    • Fiber
  • Mode Structure Based Classification:
    • Single-mode: Single EMW/Light propagated via the center for Long Haul use
    • Multi-mode: Multiple Light Rays are sent at angles for Short Range use
  • Refractive Index Distribution Based CLassification:
    • Step Index
    • Gradient Index
  • Material Based Classification:
    • Glass
    • Polymer
    • Semiconductor
• Dispersion: A phenomenon that occurs when some light travels in the fiber cladding compared to most of the lights that travel in the fiber core. Fiber Optic Cable comprises *Jacket/Sheathing/coatingg, Additional Protective Layer, Cladding, and Core*.

2. Basics of Optics

• Optics: Concerned with light and its behavioral patterns and properties.
• Major Branches:
  • Geometrical Optics: The Study of light rays
  • Physical Optics: The Study of light as waves (EMWs)
    • Speed of Light in Vacuum as an EMW: $3\times10^8 m/s$
  • Quantum Optics: The Study of Light as particles (Photons). A photon carries energy proportional to the radiation frequency but has zero rest mass.
    • Photon Energy in $eV$
• Lens: A transparent optical substance/device designed for concentrating/focusing or dispersing light rays.
• Types of Lens:
  • Concave: curves inwards (thinner in the middle and thicker at the edges) and is a diverging lens
    • Use: Glasses, Telescope, Spy Holes in Doors, ...
  • Convex: curves outward (thicker in the middle and thinner at the edges) and is converging lens
    • Use: Camera, Overhead Projector, Microscope, Magnifying glass, Simple Telescope, ...
  • Plano: A flat lens rathern than curved. Unlike concave or convex lenses, it refracts light rays but doesn't focus them.
    • For correcting astigmatism; most values will be 0.00 to +/-20.00
    • Used when one doesn't have distance problems.
• Focal Length (F): the distance of a focus (imaging point) from the surface of a lens or curved mirror. Or it is the distance between the focal point (or imaging point) and the optical center. It determines when the lens is focused at infinity
• Effective Camera Focal Length (EFL): The standard focal length is what the manufacturer writes on the lens. But the crop factor affects it. So, the effective focal length is the actual resulting value that the lens has on the camera with the crop factor adjustments.
• Optical Axis: The straight line passing through the geometrical centre of a lens and joining the two centres of curvature of its surfaces. Sometimes the optical axis of a lens is called its principal axis.
• Huygens Principle: Every point on a wavefront is itself a source of secondary spherical wavelets.
• Law of Reflection: the angle of reflection equals the angle of incidence on a smooth or highly polished surface. Keywords: Incident ray, angle of incidence, reflected ray, angle of reflection, and Normal

$$\theta_r\ =\ \theta_i$$

• Snell's Law $\rightarrow$ Law of Refraction: The law of refraction states that the incident ray, the refracted ray, and the normal to the interface, all lie in the same plane, and are related to one another by using Snell's Law. For two media with refractive indices $n_1$ and $n_2$, and incident angle $\theta_1$ and refracted angle $\theta_2$, respectively, the Snell's Law is
  • Light Ray Rare To Denser: Refracted towards the Normal
  • Light Ray Denser To Rare: Refracted away from the Normal

$$\frac{sin\theta_1}{sin\theta_2} = \frac{n_1}{n_2}$$$$n_1 sin\theta_1 = n_2 sin\theta_2$$

• Refractive Index (Index of Refraction): A value calculated from the ratio of the speed of light in a vacuum to that in a second medium of greater density.

$$n = \frac{speed\ of\ light\ in\ vacuum}{speed\ of\ light\ in\ other\ medium}$$

• Critical Angle: The greatest angle at which a ray of light, travelling in one transparent medium, can strike the boundary between that medium and a second of lower refractive index without being totally reflected within the first medium.

$$\theta_c = sin^{-1}(\frac{n_2}{n_1})$$

• Total Internal Reflection: complete reflection of a ray of light within a medium such as water or glass from the surrounding surfaces back into the same medium.
  • It occurs when the incident angle on a denser medium is greater than the critical angle.

$$\theta_i > \theta_c$$

• Bragg Reflection Law: this law simply states that when the X-ray is incident onto a crystal surface, its angle of incidence is $\theta$ and it is reflected back with the same angle of scattering $\theta$. And if the path difference $d$ is equal to a whole number $n$ of wavelength, constructive interference will occur.

  • Bragg's equation For Constructive Interference:

  $$nλ=2dsinθ$$

  • For Volume Bragg Grating:

  $$2n\Lambda sin(\theta + \phi ) = \lambda_B$$

• Collimation: the accurate adjustment of the line of sight of a telescope. Type of adjustment, readjustment, or registration. The act of adjusting something to match a standard.

  • (of rays of light or particles) made accurately parallel. "a collimated electron beam"

• Calibration: the process of configuring an instrument to provide a result for a sample within an acceptable range. Eliminating or minimizing factors that cause inaccurate measurements is a fundamental aspect of instrumentation design.

• Thin Lens Equation:

$$\frac{1}{f} = \frac{1}{s_o} + \frac{1}{s_1}$$

• Field of View (FOV): Extent of observable world wrt a certain point or source of light. It measures a linear distance
  • In the eyes of optical instruments or sensors: a solid angle through which a detector is sensitive to electromagnetic radiation.
  • HFOV: Horizontal FOV --> Linear Horizontal view of sensor width
  $$HFOV = 2F\times tan(\frac{\theta_h}{2})\times \frac{\pi}{180}$$ - HFOV of Human Vision = $154^o$
  • VFOV: Linear Vertical view of sensor Height
  $$VFOV = 2F\times tan(\frac{\theta_v}{2})\times\frac{\pi}{180}$$
  • DFOV: Linear Diagonal view of sensor
  $$DFOV = \sqrt(HFOV)^2 + (VFOV)^2$$ - Current AR Smart Glasses' DFOV is about $50^o$
• Angular Field of View (AFOV): The Angle between two light rays in simple terms. AFOV relates to angles or angular distance unlike FOV.

$$AHFOV = \theta_h = 2\times tan^{-1}(\frac{W}{2F})\times \frac{180}{\pi}$$$$AVFOV = \theta_v = 2\times tan^{-1}(\frac{H}{2F})\times \frac{180}{\pi}$$

• Angular Resolution: The ability of any image-forming device such as an optical or radio telescope, a microscope, a camera, or an eye, to distinguish small details of an object, thereby making it a major determinant of image resolution.
  • Human Eye Resolution Limit = $1.2$ arc min or around $50$ pixels/degree.
  • An AR Glass that gives an angular resolution of 1.2 arc min and an HFOV of $154^o$ is indistinguishable from reality.
• Pixel Size: Size of a single picture element (photosite or well) or pixel on an image sensor, usually in mm or $\mu m$.
• Pixel Angle: Angle of a single pixel. A single pixel typically corresponds to 0.01 to 0.03 visual degrees.

$$PixelAngle = 2\times tan^{-1}(\frac{PixelSize\ \times\ BinningSize}{2\times EFL})$$

• Binning: The process of downsampling an image by creating a set of super pixels. It is a tradeoff between read-noise and spatial resolution.

  • Horizontal, Verticall, or Both
  • CCD Sensor: Bucket Brigade $\rightarrow$ Photosite Noise
    • Picture element/Pixel/Photosite/Well
      • $\mu lens$
      • Color Filter
      • Well
  • CMOS Sensor: Parallel Processing
  • Symetrical Binning: $1\times1$, $2\times2$, $3\times3$, $4\times4$
  • ADC $\rightarrow ADU$: 16/14/12 Bits
  • Upside/pros: Computational Efficiency, Increased Light Sensitivities, and Reduced Read Noise or Better SNR
  • Downside/Cons: Reduced Spatial Resolution, and less details (or poor angular resolution)

• OLED: The acronym 'OLED' stands for Organic Light-Emitting Diode - a technology that uses LEDs in which the light is produced by organic molecules. These organic LEDs are used to create what are considered to be the world's best display panels.

3. AR Waveguides

• There are two types of technologies that help bring the images to the eyes via smart AR glasses.
  1. Curved Visors (Not convenient)
     • Large, Micro, or LCoS Display
     • Projector
  2. Waveguides (Integrates Imaging Relay Tehniques and is convenient):
     • Diffractive Outcoupling
     • Holographic Outcoupling
     • Polarized Thin Layer Outcoupling
     • Reflective Outcoupling
• Broadly WG are categorized as:
  • Diffractive and
    • Diffractive WG
    • Holographic WG
  • Geometric
    • Polarized WG
    • Reflective WG
• General Optical Concept of how VR Headsets Work

Image("./Figs/wg2.png")

        Fig 2: Optical Design Of Exsiting/Old Headset VRs

Image("./Figs/wg3.png")

Fig 3: Projection of collimated virtual Image Via WG From An OLED or $\mu LED$ Display To An Eyebox/Output Grating using a flat mirror.

Image("./Figs/wg4.png")

Fig 4: Projection of collimated virtual Image Via WG From An OLED or $\mu LED$ Display To An Eyebox/Output Grating using a curved mirror.

Image(("./Figs/wg5.png"))

                                      Fig 5: VH-FOV

3.1 WG With Diffractive Outcoupling

• A WG Based on Display with surface relief where the image is essentially sliced into a series of vertical stripes and reassembled in the eye.
• Upsides:
  • Easy to make for small FOV
  • Used in Nokia, Vuzix, and Microsoft Hololens
• Downsides:
  • More slanted, more difficult to make because index matching can be difficult. Mostly based on polymer imprint.
  • Creates Rainbow Effect --> Monochrome is better
  • Higher FOV, Higher coupling angle, and higher color non-uniformity. Beyond $20^o$, the color non-uniformity becomes very noticeable.

Image(("./Figs/wg6.png"))

                Fig 6: Diffractive WG (Nokia Surface Relief WG)

3.2 WG With Holographic Outcoupling

• Upsides:
  • Similar to Diffractive WG but doesn't need the ridges instead a curved photopolymer produces the diffraction
  • New Materials can increase the refractive index modulation and improve this (Chemistry of the photopolymer)
• Downsides:
  • Loses Intensity with Angular Variation and Color non-uniformity --> limited FOV
  • Color cross-talks becuase3 layers sandwitched halograms need to be used. That is why you see a halo around images

Image(("./Figs/wg7.png"))

                      Fig 67: Holographic WG

3.3 WG With Polarized Reflective Outcoupling

• Employs multilayer coating and embedded polarized reflectors
• Upsides:
  • Large FOV, large eye motion box
  • Less color non-unifomrity
• Downsides:
  • Low efficiency: the refectivity of the polarized refelectors are quite small becuase it is less than $\frac{1}{n}$ (n is the numbe rof reflectors). Example: for 6 reflectors used in current light guide, the reflectivity is around 10%.
  • The polarized coatings are multilayer coatings of 25-30 layers each and must be deposited on the glass as plastic is not compatibe with this process. The layers should be glued with extremely high accuracy --> Not easy for high volume production.
  • System and reflectors are polarized which makes the system inherently inefficient because nearly 70% of the light is lost when it is reflected.
• Example: Early Lumus WG

Image(("./Figs/wg8.png"))

                Fig 8: Polarized Reflective WG (Early Lumus Prototype WG)

3.4 WG With Reflective Outcoupling

• The light is extracted using a semi reflective mirrored structure using traditional coating s found throughout the optics industry.
• Upsides:
  • Doesn't suffer from the color non-uniformity issues
  • Compatible with molded plastic subsrate and traditional coating techniques (Early Epson Maverios Prototype)
• Downsides:
  • Very thick (1cm), distorts the background
  • Giant slab infront of your eyes

WG With Thin Reflective Outcoupling

• Instead of $\mu$ ridges they use macro ridges with mm sized steps
• Upsides:
  • More efficient larger FOV and eyebox and no color non-uniformity issue.
  • Monolithic Light Guuide (out of one piece of plastic) which is coated with a semi reflective coating
• Downsides:
  • Moulding the light guide and its surface structure precisely is difficult
  • Possible distortion in the image or background due to thickness (5mm)

Image(("./Figs/wg9.png"))

                Fig 9: Reflective WG 

4. Major Components of Smart AR Glasses

• Major Components of an AR Glass Eye Wear:

  • Glasses Frame: Holds the whole set of components together.

  • Battery (typically LiPo Battery): Source of Electrical power

  • $\mu$LED Display (DPA) $\rightarrow$ Source of Virtual Image. It generates the virtual image.

  • Optical Engine/Projector Module: Performs Light Beam Collimation and Projects the virtual image produced by the $\mu$LED towards the entrance aperture of the WG.

    • Small Entrance Aperture: Enabls Compact Projector Module

  • WG: performs 2D pupil expansion and propagates the image towards the Input Grating.

    • 2D RWG enables very large DFOV in a $1:1$ aspect ratio
    • Could be Diffractive (DWG) or Reflective (RWG)

  • Input Grating: Receives the projected lights (the projected & collimated virtual image) and sends it to Fold Grating.

  • Fold Grating: Receives light rays reflected by the Input Grating via the WG and send them to the eyebox or output grating.

  • Output Grating/Eyebox: Sends the virtual image in parralel to the eye balls. The light rays come to our eye balls in parallel; as a result, the image appears as if at infinity.

    • Embedded Facets on the eyebox expand the image and project it to the eye.
    • It makes the virtual image appear right infront of our eye balls.
    • this is the place where you can see focused image

  • Lens: The most complicated technology is in the lenses for both dispersing and focusing light rays

  • Camera: For taking pictures

  • Bluetooth: For communication with Mobiles, Laptops, or other blutooth protocol supporting devices

Image(("./Figs/wg10.png"))

                             Fig 10: Eyebox/Output Grating On The Glass Frame

Image(("./Figs/wg11.png"))

  Fig 11: Nano Precision Optical Structures (Input Grating, Fold Grating, and Output Grating)

Standard Manufacturing Processes of an AR Waveguide

1. Coating: Input material is often a simple BK7 Plates of glass. They are put through a massive coding machine, where some secret codes are added (A company's Secret Coca Cola Recipe ... haha)
2. Stacking: Coded Glasses are stacked together as per design
3. Slicing: Slicing off segements across multiple planes. It is like a wafer slicing employed in semiconductors
4. Lap & Polish: fogy sliced glasses are put through lap and polish machine to bring them down to a nice and clean optical grade
5. Shaping: shape the glasses for the particular eye wear need or design.
6. Testing: perform IQT and MTF tests to verify its quality

• BK7 is a high quality optical glass that is used whenever the additional benefits of fused silica are not required. Since BK7 performs well in all chemical tests, and no additional or special handling is required, costs of manufacturing are reduced.
• Costing:
  • Consumables Cost
  • Labor Cost
  • Maintenance and Repair Cost
  • Facility and Utility Cost

Challenges

1. Ensuring that all the Nano Precision Optical Structures called the Input Grating, the Fold Grating, and the Output Grating are all uniform and consistent.
2. Literally, there are millions of light paths resulting from the meeting intersections of the image beam with the grating fringes that comprises these structures.
3. The nature of the high precision bragg grating based diffractive optics leads to non-uniformities due to gaps or overlaps of the image beams propagating down the WG.
4. Imperfections in the gratings and nonplanarity of the WG substrate also cause nonuniformity.

Reverse Ray Tracing

• Out of the millions of possible beam interactions, only a subset of them contribute to the image sent to the eyebox.

• Hence, through the application of a Reverse Tracing on the image light rays on the user eyebox, it is possible to pin point the exact ouput Fold and Input Diffractive Grating Regions that actually contribute to the images seen from the within the eyebox.

• An example that shows how corrections by way of Reverse Ray Tracing and advanced optics are illustrated in the figure below.

        1 2 3
   IG   4 5 6
        7 8 9            9 8 7
                      OG 6 5 4
                9 6 3    3 2 1
             FG 8 5 2
                7 4 1
  

Image(("./Figs/wg12.png"))

Fig 11: Reverse Light Ray Tracing To Determine The Subsets of Rays Contributing to The Formation of the V-Img On The Eyebox

5. Color Spaces

• Color space: It describes a specific, measurable, and fixed range of possible colors and luminance values. Its most basic practical function is to describe the capabilities of a capture or display device to reproduce color information.
• Colorimeter: an instrument for measuring the intensity of color.
• Tristimulus Values: measure light intensity based on the three primary color values (RGB), typically represented by $X, Y, \& Z$ coordinates. The tristimulus values system is the foundation of color language, also referred to as the CIE color system, and is used to communicate precise color values around the world.
• $CIE_{xy}Y$ Color Space: $Y$ is the Relative Luminance and $x&y$ are the derived params that specifiy the chromaticity.
• $XYZ$: Tristimulus Values

$$Y = from\ CIE_{xy}Y\ Color\ Space\ Measurement$$$$Y = \frac{Y}{y}x$$$$Z = \frac{Y}{1-x-y}$$

• $xyz$: Trichromaticity Values

$$x = \frac{X}{X+Y+Z}$$$$y = \frac{Y}{X+Y+Z}$$$$z = \frac{Z}{X+Y+Z} = 1-x-y$$

• Trichromaticity Summation Rule:

$$x+y+z = 1$$

• CIE 1976 L, u, v* color space: commonly known by its abbreviation CIELUV, is a color space adopted by the International Commission on Illumination (CIE) in 1976, as a simple-to-compute transformation of the 1931 CIE XYZ color space, but which attempted perceptual uniformity.
• Wavelengths of RGB colors:
  • $Red\rightarrow 700nm$: Long waves with $f=430THz$, & 1.8eV energy
  • $Green\rightarrow 541.1nm$: Short Wave with $f=600THz$, & 2.38 eV
  • $Blue\rightarrow 435.8nm$: Short Wave with $f=750THz$, & 3.1eV energy
  • Shorter waves vibrate at higher frequencies and have higher energies
  • The photon energy of visible light ranges from 2 to 2.75 electron volts (eV)
    • Blue Light Hurts The Eye because it has higher energy, 3.2eV

6. AR Glasses Test Requirements

• Wide Field of View (FOV): the glasses must have an FOV large enough to cover the natural eye HFOV, VFOV, or DFOV and Angular Resolution. Measure the FOV of the glasses during testing.
• Excellent Color Fidelity: Check using the RGB GamutTarget during testing. Color accuracy Test is performed.
• High Luminance: RGB Luminance should be high enough to the point comfortable to the eyes
• Uniformity: Make sure that the display unit has a uniform look no matter wherever the user is looking at within the eyebox display.
• Contrast: The higher the contrast ratio, the better the quality if the display. In the traditional RYB color model, the complementary color pairs are red–green, yellow–purple, and blue–orange. It is computed as the ratio of the color with highest value to the color with lowest value value. Often B/W checkerboard pattern images are employed.
• Sharpness: considered a significant factor in determining image quality as it is the factor that determines the amount of detail an imaging system can reproduceUsing
  • Image sharpness can be measured by the “rise distance” of an edge within the image. With this technique, sharpness can be determined by the distance of a pixel level between 10% to 90% of its final value (also called 10-90% rise distance
  • measured using Modulation transfer Function (MTF)/ Spatial Frequency Response (SFR)

7. Conclusion

• "The Success of smart AR Glasses in a consumer acceptable form-factor begins and ends with near-eye optics" Christopher Grayson

                         "With Patience We Can Dissect An Ant And Reach Its Heart"
                                                  ~END~